Skip to content

HADOOP-19184. S3A Fix TestStagingCommitter.testJobCommitFailure#6843

Merged
mukund-thakur merged 2 commits intoapache:trunkfrom
mukund-thakur:HADOOP-19184
May 28, 2024
Merged

HADOOP-19184. S3A Fix TestStagingCommitter.testJobCommitFailure#6843
mukund-thakur merged 2 commits intoapache:trunkfrom
mukund-thakur:HADOOP-19184

Conversation

@mukund-thakur
Copy link
Contributor

@mukund-thakur mukund-thakur commented May 22, 2024

Follow up of HADOOP-18679

Description of PR

How was this patch tested?

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@mukund-thakur
Copy link
Contributor Author

mukund-thakur commented May 22, 2024

not yet ready. will clean this up and retest.

@hadoop-yetus
Copy link

hadoop-yetus commented May 22, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 7m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 32m 10s trunk passed
+1 💚 compile 0m 24s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 0m 19s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 0m 21s trunk passed
+1 💚 mvnsite 0m 24s trunk passed
+1 💚 javadoc 0m 19s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 22s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 0m 42s trunk passed
+1 💚 shadedclient 19m 46s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 19s the patch passed
+1 💚 compile 0m 20s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 0m 20s the patch passed
+1 💚 compile 0m 15s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 0m 15s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 12s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚 mvnsite 0m 19s the patch passed
+1 💚 javadoc 0m 9s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 17s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 0m 43s the patch passed
+1 💚 shadedclient 19m 37s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 20s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
90m 24s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/1/artifact/out/Dockerfile
GITHUB PR #6843
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux b31e20ea9151 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / bdbc61d
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/1/testReport/
Max. process+thread count 559 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

steveloughran commented May 23, 2024

I can only trigger the failure on trunk in a parallel test run! this explains why yetus missed it.

except: I can also trigger it from the IDE, which isn't parallel.

I don't understand this. Except to note that with this patch, even parallel runs are happy.

@mukund-thakur
Copy link
Contributor Author

mukund-thakur commented May 23, 2024

Tested using us-west-1 bucket. All good.

@hadoop-yetus
Copy link

hadoop-yetus commented May 23, 2024

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 24s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 39m 55s trunk passed
+1 💚 compile 0m 32s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 compile 0m 26s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 checkstyle 0m 23s trunk passed
+1 💚 mvnsite 0m 32s trunk passed
+1 💚 javadoc 0m 21s trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 26s trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 0m 52s trunk passed
+1 💚 shadedclient 28m 4s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 24s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javac 0m 26s the patch passed
+1 💚 compile 0m 20s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 javac 0m 20s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 javadoc 0m 13s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚 javadoc 0m 20s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚 spotbugs 0m 51s the patch passed
+1 💚 shadedclient 27m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 16s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 27s The patch does not generate ASF License warnings.
109m 27s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/2/artifact/out/Dockerfile
GITHUB PR #6843
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux c82220bdc649 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 2485212
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/2/testReport/
Max. process+thread count 706 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6843/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. If we did more mocking we could do something more sophisticated/reusable, but it'd still be really brittle...

@steveloughran steveloughran changed the title HADOOP-19184. Fix UT TestStagingCommitter.testJobCommitFailure HADOOP-19184. S3A Fix TestStagingCommitter.testJobCommitFailure May 27, 2024
@mukund-thakur
Copy link
Contributor Author

mukund-thakur commented May 28, 2024

I was trying to do the mocking of the whole S3AStore but was unsuccessful. It was getting complicated without a working solution. maybe we can revisit it later. I think it will be required in the future.

@mukund-thakur mukund-thakur merged commit f4fde40 into apache:trunk May 28, 2024
mukund-thakur added a commit that referenced this pull request May 28, 2024
@mukund-thakur
HADOOP-19184. S3A Fix TestStagingCommitter.testJobCommitFailure (#6843)
183a1f1 
Follow up on HADOOP-18679

Contributed by: Mukund Thakur
@steveloughran
Copy link
Contributor

steveloughran commented May 29, 2024

w.r.t mocking we'd have a S3AFilesystem.createStore() method to create it, the mocking could do its work there, maybe by invoking superclass then mocking the operations it needs to.

@mukund-thakur
Copy link
Contributor Author

mukund-thakur commented May 29, 2024

w.r.t mocking we'd have a S3AFilesystem.createStore() method to create it, the mocking could do its work there, maybe by invoking superclass then mocking the operations it needs to.

Yes, something like that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steveloughran steveloughran steveloughran approved these changes

Assignees

No one assigned

Labels

AWS TOOLS trunk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mukund-thakur @hadoop-yetus @steveloughran